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STUDY  REPORT  ABSTRACT: 

The  purpose  of  this  report  was  to  assess  the  current  role  of  operational  test 
and  evaluation  (OT&E)  and  the  Air  Force  Test  and  Evaluation  Center  (AFTEC) 
in  the  AF  aircraft  acquisition  process.  Additionally,  it  was  intended  to 
provide  recommendations  to  the  program  manager  (PM)  on  ways  to  ensure 
adequate  coverage  of  OT&E  requirements  in  his  program. 

A trace  of  the  OT&E  policy  in  DOD  and  AF  from  1970  until  1976  leads  to  an 
analysis  of  the  new  DOD  draft  Test  and  Evaluation  Directive  5000.3,  dated 
7 March  77.  This  revision  to  DOD  T&E  policy  emphasizes  earlier  operator 
involvement  and  combined  development  and  operational  testing  in  the 
acquisition  process. 

A series  of  interviews  with  DOD  and  AF  officials  was  used  to  compare  the 
current  perceptions  of  the  role  and  effectiveness  of  lOT&E  and  AFTEC.  The 
AFTEC  view  of  itself  is  compared  with  the  other  viewpoints  expressed  in 
the  interviews.  While  differences  exist,  there  is  an  overall  similarity 
in  perception. 

Finally,  the  paper  makes  recommendations  to  the  PM  as  to  how  he  can 
better  ensure  that  he  includes  all  valid  operational  requirements  in  his 
test  program.  The  areas  covered  are  establishing  the  baseline,  organizing 
and  planning  with  primary  emphasis  on  the  latter. 
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EXECUTIVE  SUMMARY 


The  purpose  of  this  paper  was  two-fold:  first,  to  investigate  the 
evolution  of  operational  test  and  evaluation  (OT&E)  policy  in  the 
Air  Force's  aircraft  acquisition  process  and  in  particular  the  role  of 
the  Air  Force  Test  and  Evaluation  Center  (AFTEC) ; second,  to  provide 
recommendations  to  the  program  manager  (PM)  on  how  to  ensure  that  he 
includes  valid  OT&E  requirements  in  his  program. 

The  methodology  used  was  to  trace  DOD  and  AF  policy  on  OT&E  from 
1970  to  the  present  with  special  emphasis  on  an  analysis  of  the  OT&E 
portions  of  the  draft  DOD  Directive  5000.3,  "Test  and  Evaluation", 
dated  7 March  1977.  This  was  followed  by  a series  of  interviews  intended 
to  provide  a comparison  of  the  views  of  AFTEC  with  those  of  other  elements 
of  DOD  and  the  AF  on  the  Implementation  and  effectiveness  of  lOT&E  and 
AFTEC.  Finally,  a series  of  recommendations  to  the  PM  were  made  relative 
to  establishing  baselines,  organizing  and  planning  for  his  program  to 
ensure  proper  coverage  of  OT&E  requirements. 

The  significant  findings  include  the  fact  that  the  new  5000.3  will 
emphasize  combined  development  and  operational  testing  whenever  possible 
prior  to  production  decision  and  an  earlier  involvement  of  AFTEC  and  the 
user  in  the  development  cycle.  The  interviews  determined  that,  despite 
initial  conflict,  there  is  now  generally  good  agreement  within  DOD  and 
the  AF  on  the  role  of  AFTEC.  There  is  also  a need  to  rewrite  AFR  80-14 
to  bring  it  into  line  with  the  new  5000.3  and  to  further  clarify  the 
management  relationships  between  the  PM  and  AFTEC. 


This  paper  should  be  useful  to  PMs  of  large  or  small  programs  in 
structuring  their  test  programs  to  achieve  the  proper  balance  between 
developmental  and  operational  tests.  It  can  also  serve  to  provide 
both  developer  and  operator  personnel  with  a better  understanding  of 
the  interrelationship  of  their  roles  in  the  acquisition  process. 
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SECTION  I 


INTRODUCTION 

Back-Rround 

Flight  test  of  powered,  heavier-than-air  aircraft  has  been  going 
on  in  the  United  States  since  17  December  1903,  when  the  Wright  Brothers 
flew  what  was  probably  the  shortest  successful  test  flight  on  record. 

In  February,  1977,  the  Space  Shuttle,  piggybacked  on  a Boeing  747, 
made  its  first  test  flight.  Both  events  were  uniquely  exciting.  The 
former  took  the  effort  of  two  extraordinary  men,  the  latter  that  of 
thousands  of  dedicated  men.  Between  these  two  aviation  milestones, 
hundreds  of  aircraft  have  gone  through  extensive  flight  test  programs 
with  varying  degrees  of  success.  While  test  program  success  has  not 
necessarily  resulted  in  the  production  of  a new  aircraft,  an  unsuccessful 
test  program  just  about  guaranteed  that  the  plane  would  not  be  produced. 
In  other  words,  the  test  program  can  lead  to  the  termination  of  an 
acquisition  program.  If  the  test  program  Identified  a major  fault  that 
led  to  program  cancellation,  the  test  program  had  accomplished  its 
purpose  just  as  well  as  if  it  verified  that  the  system  under  development 
was  ready  for  production.  If,  on  the  other  hand,  significant  problems 
surface  after  the  new  aircraft  is  in  the  operational  units,  the  Air  Force 
is  accused  of  fielding  an  underdeveloped  system.  Conversely,  as  the 
time  between  Milestone  0 (Program  Initiation  under  DOD  Directive  5000.1 
dtd  18  Jan  1977)  and  Initial  Operational  Capability  (IOC)  grows  longer 
and  more  expensive,  the  AF  is  accused  of  testing  to  obsolescence.  The 
pendulum  swings  back  and  forth.  It  is  the  responsibility  of  the  program 
manager,  the  roan  charged  with  the  development  of  the  system,  to  ensure 
that  he  plans  and  Implements  an  efficient  and  effective  test  progr^. 


The  purpose  of  this  paper  is  to  investigate  the  evolution  of 
operational  test  and  evaluation  (OT&E)  within  the  Department  of  Defense 
(DOD)  and  the  Air  Force  and  to  provide  recommendations  to  the  program 
manager  (PM)  that  will  help  him  ensure  that  valid  OT&E  requirements  are 
included  in  his  program.  Emphasis  will  be  placed  on  initial  operational 
test  and  evaluation  (lOT&E),  that  portion  of  OT&E  accomplished  prior 
to  the  initial  production  decision,  and  the  role  of  the  Air  Force 
Test  and  Evaluation  Center  (AFTEC) . The  methodology  employed  is  to 
trace  DOD  and  AF  policy  on  T&E  since  1970  with  particular  attention 
given  to  an  analysis  of  the  7 March  1977,  DOD  Directive  5000.3  (DRAFT). 
Through  a series  of  interviews,  the  paper  compares  the  views  of  various 
DOD  and  AF  officials  on  lOT&E  policy  and  AFTEC  with  the  views  of  the 
AFTEC  commander.  Finally,  recommendations  are  made  as  to  how  the  PM 
might  better  ensure  that  valid  OT&E  requirements  are  incorporated  in  his 
program  plans. 

The  scope  of  the  paper  is  limited  to  the  Air  Force,  to  major 
aircraft  acquisition  programs  and  to  the  testing  prior  to  the  system 
production  decision. 

Definitions 

To  preclude  any  confusion,  development  test  and  evaluation  (DT&E) 
and  OT&E  are  defined  as  per  DRAFT  DOD  Directive  5000.3  dtd  7 March  1977. 

a.  DT&E  is  that  test  and  evaluation  conducted  to  assist  the 
engineering  design  and  development  process  and  verify  attainment 
of  technical  and  performance  specification  and  objectives.  DT&E 

is  normally  managed  and  monitored  by  the  DOD  components'  developing 
agency. . . 

b.  OT&E  is  that  test  and  evaluation  to  estimate  a prospective 
system's  operational  effectiveness  and  operational  suitability... 
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SECTION  II 


OT&E  POLICY  1970-1976 

The  policies  gcverning  T&E  have  changed  over  the  years  in 
response  to  changing  needs  and  a variety  of  Internal  and  external 
criticisms.  Since  1970,  the  majority  of  these  policy  changes  and 
criticisms  have  been  directed  toward  the  operational  aspects  of  testing. 

In  1970  the  flight  test  portion  of  the  aircraft  acquisition  process 
was  separated  into  three  categories.  Category  I,  Subsystem  DT&E,  was 
generally  contractor  conducted  and  was  totally  developmental  in  nature. 
APR  80-14  defined  CAT  I as  "development  testing  and  evaluation  of  the 
individual  components,  subsystems,  and  in  certain  cases  the  complete 
system.  (9:4)  ^ There  were  Air  Force  Preliminary  Evaluations  but  these 
too  were  DT&E  oriented.  In  Category  II,  System  DT&E,  the  players 
changed  somewhat  but  the  emphasis  did  not.  The  Air  Force,  through 
Air  Force  Systems  Command  (AFSC),  had  the  lead  role  with  the  contractor 
still  heavily  involved  but  in  a secondary  role.  AFR  80-14  defined 
CAT  II  as  "test  and  evaluation  spanning  the  Integration  of  subsystems 
into  a complete  system  in  as  near  an  operational  configuration  as 
practicable."  The  using  command  became  involved  during  CAT  II  by 
assigning  the  Deputy  Test  Director  and  through  participation  in  "hands- 
on"  maintenance  with  the  contractor  at  the  test  site.  However,  CAT  II 
was  seldom  "operational  in  nature".  (3:26)  The  production  decision 


This  notation  will  be  used  throughout  the  report  for  sources  of 
quotations  and  major  references.  The  first  number  is  the  source  listed 
in  the  bibliography.  The  second  number  is  the  page  in  the  reference. 
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1 for  the  system  was  generally  made  during  CAT  II.  Category  III,  System  j 

I OT&E,  was  the  responsibility  of  the  using  command.  It  included  "... 

I all  components,  support  items,  personnel  skills,  technical  data  and 

procedures ...  under  as  near  operational  conditions  as  practicable."  (9:5)  j 

I Since  the  tests  were  conducted  on  the  production  item  after  delivery 

j to  the  using  command,  "such  testing  was  to  determine  how  best  to  operate 

a system,  rather  than  any  endeaver  to  provide  data  for  decision  makers 
* to  use  in  determining  whether  or  not  to  acquire  a system."  (21:85) 

The  above  was  the  situation  when  the  Blue  Ribbon  Defense  Panel  (BRDP) 

! 

made  its  Report  to  the  President  and  the  Secretary  of  Defense  on  1 July 
: 1970.  The  Executive  Summary  statement  relative  to  OT&E  was  direct  and 

: to  the  point.  "Operational  test  and  evaluation  has  been  too  infrequent, 

poorly  designed  and  executed,  and  generally  inadequate."  (2:2)  On  the 
other  hand  DT&E  was  "well  understood  and  faithfully  executed ... (and) .. . 

I 

not  considered  to  be  a major  problem  area."  (2:88)  The  report  went  on  to 

i 

point  out  that  OT&E  was  not  just  physical  testing  but  that  "To  be  effective,  J 

OT&E  must  be  a total  process,  using  all  appropriate  methods  of  evaluation, 

■•i 

which  spans  the  entire  cycle  of  a system  from  initial  requirement  until  j 

i it  is  phased  out  of  the  operational  forces."  (2:88)  < 

I 1 

Referring  specifically  to  the  Air  Force,  the  panel  stated:  i 

There  are  three  principal  problems  with  Air  Force  OT&E,  as 
currently  done.  First,  operational  considerations  receive  much 
too  little  attention  in  Categories  I and  II.  Second,  the 

operational  commands  responsible  for  Category  III  and  Operational  ' 

Employment  Testing  lack  both  the  personnel  and  facilities  to  be 

effective.  Finally,  all  of  the  categories  are  too  duplicative  j 

and  time-consuming.  (2:90)  i 

i 


The  panel  also  made  sweeping  recommendations  on  organization  within 
the  Office  of  the  Secretary  of  Defense  (OSD).  The  most  significant 
for  the  purposes  of  this  paper  concerned  the  independent  OT&E  agency. 

The  wording  is  somewhat  indirect  but  the  meaning  is  clear.  "The  Secretary 
of  Defense  should  communicate  to  the  Military  Departments ...  his  conviction 
the  cause  of  effective  OT&E  is  best  served  when  independent  OT&E  organiza- 
tions report  directly  to  the  Chiefs  of  Service,  Service  Secretaries  or 
both."  (3:9) 

DOD  response  was  immediate  and  positive.  SecDef  Laird  asked  the 
services  to  forward  their  individual  proposals  for  meeting  OT&E  require- 
ments to  him  not  later  than  1 September  1970.  (17:1)  The  results  were 
inconclusive  and  resulted  in  a memo  from  Deputy  SecDef  Packard  on 
11  February  1971.  He  wanted  an  OT&E  "...agency  which  is  separate  and 
distinct  from  the  developing  command  and  which  reports  the  results  of 
its  T&E  efforts  directly  to  the  Chief  of  Service."  (18:1) 

The  Air  Force  response  did  not  establish  the  independent  organization 
but  it  did  make  significant  changes.  The  concept  of  categories  was 
eliminated  and  replaced  with  the  DT&E/OT&E  philosophy.  Also  Introduced 
was  initial  operational  test  and  evaluation  to  be  conducted  prior  to 
the  production  decision.  More  active  participation  by  the  using  and 
supporting  commands  was  Incorporated  and  the  user  was  required  to 
provide  an  lOT&E  report  to  the  production  decision  process.  (19) 

Parallel  with  the  memorandum  - response  cycle,  the  first  DOD 
Directive  5000.1,  dated  July  31,  1971,  "Acquisition  of  Major  Defense 
Systems"  was  prepared  and  released.  Its  purpose  was  to  provide  formal 


acquisition  policy  within  DOD  and  to  establish  the  Defense  Systems 

Acquisition  Review  Council  (DSARC)  and  the  Development  Concept  Paper 

(DCP)  for  major  programs.  No  mention  of  direct  user  involvement  prior 

to  DSARC  II  (Full-Scale  Development  decision)  was  made.  Nor  was  there 

any  requirement  for  an  independent  operational  test  agency.  It  should 

be  noted  that  this  directive  followed  by  five  months  Secretary  Packard's 

memo  calling  for  an  independent  agency.  However,  it  was  very  explicit 

on  the  role  and  Importance  of  the  operational  assessment: 

Test  and  evaluation  shall  commence  as  early  as  possible.  A 
determination  of  operational  suitability,  including  logistic 
support  requirements,  will  be  made  prior  to  large-scale  pro- 
duction commitments,  making  use  of  the  most  realistic  test 
environment  possible  and  the  best  representation  of  the  future 
operational  system  available.  The  results  of  this  operational 
testing  will  be  evaluated  and  presented  to  the  DSARC  at  the 
time  of  the  production  decision.  (12:5) 

The  Air  Force  had  already  incorporated  both  the  spirit  and  the 
intent  of  5000.1  through  a Chief  of  Staff  letter  on  OT&E  on  1 Apr  71 
which  called  for  "operational  test  and  evaluation ...  conducted  by 
operational  forces  to  determine  operational  effectiveness  and  suitability 
..."  (21:89)  The  reporting  channel  for  the  test  results  was  Independent 
of  the  developer  and  the  AF  felt  that  this  complied  with  the  intent  of 
the  DOD  guidance  and  the  BRDP  recommendations. 

It  was  not  until  a year  and  a half  later,  19  Jan  1973,  that 
specific  DOD  policy  on  T&E  was  formalized  into  DODD  5000.3,  "Test  and 
Evaluation".  The  Directive  confirmed  the  existing  policy  that  OT&E 
would  be  conducted  by  "operational  and  support  personnel".  (15:3)  lOT&E 
was  to  be  accomplished  during  Full-Scale  Development  to  assist  in 
evaluating  operational  effectiveness.  The  most  significant  requirement 
was  that  "one  major  field  agency  (or  a limited  number  of  such  major 
field  agencies)  separate  and  distinct  from  the  developing/procuring 


connnand"  be  established.  This  agency  was  to  report  directly  to  the 
Chief  of  Staff  and  "Insure  that  the  OT&E  is  effectively  planned  and 
conducted."  (15:3)  The  OT&E  was  to  be  separate  from  DT&E  but  lOT&E 
could  be  combined  with  DT&E  if  "separation  would  cause  delay  involving 
unacceptable  military  risk,  or  would  cause  an  unacceptable  increase 
in  the  acquisition  cost".  (15:4)  In  either  eventuality,  the  OT&E 
agency  was  responsible  for:  insuring  that  the  tests  were  planned  and 
executed  so  as  to  provide  the  necessary  operational  test  data;  partici- 
pating actively  in  the  tests;  and  separately  evaluating  the  test  results. 
There  was  no  mention  of  the  seeming  conflict  with  5000.1  which  stated 
that  "The  development  and  production  of  a major  defense  system  shall  be 
managed  by  a single  program  manager..."  and  the  split  OT&E  responsibility 
of  5000.3.  This  aspect  will  be  considered  further  later  in  the  paper. 

The  Air  Force  established  the  Air  Force  Test  and  Evaluation  Center 
(AFTEC)  as  a separate  operating  agency  on  1 January  1974,  thus  being 
the  last  of  the  three  services  to  have  an  independent  operational  test 
agency.  AFTEC,  with  its  200  personnel,  was  to  assess  the  operational 
effectiveness,  suitability  and  logistic  supportability  of  new  AF  systems 
and  to  report  the  results  directly  to  the  AF  Chief  of  Staff.  The  using 
command  would  continue  to  provide  personnel  to  actually  conduct  the 
operational  tests  under  AFTEC  management.  Thus  the  independent  evaluating 
and  reporting  functions  were  established  but  the  people  actually  doing 
the  work  did  not  change  significantly.  The  only  visible  change  at  the 
test  sites  was  that  an  AFTEC  individual  was  assigned  to  the  test  force 
and  given  operational  control  over  the  using  command  personnel  who  were 
already  in  place.  To  say  the  least,  the  action  created  considerable 
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I consternation  and  misunderstanding  of  the  AFTEC  role.  Joint  Test  Force 

! 

I (JTF)  directors  saw  their  "test  teams"  being  fragmented  and  their 

I 

[ authority  diluted.  It  was  difficult  if  not  impossible  to  draw  a JTF 

[ 

j 

I organizational  chart  that  really  depicted  the  organization.  A strong 

f 

I adversary  relationship  existed  as  a result  of  the  new  organization  being 

f imposed  upon  already  existing  and  smoothly  running  test  forces.  This 

i 

aspect  will  also  be  pursued  further,  later  in  the  paper. 

DODD  5000.1  went  unchanged  for  nearly  4%  years,  while  5000.3 
experienced  only  minor  changes.  The  latter  deleted  the  option  of  having 

[ 

more  than  one  independent  agency  and  further  stipulated  that  the  one 

i 

; remaining  field  agency  would  be  independent  of  the  using  command  as  well 

; as  the  developing  command.  The  22  Dec  1975  change  to  5000.1  left  intact 

1 

the  concept  of  the  single  program  manager  being  responsible  for  development 
i of  the  system  and  the  paragraph  on  T&E  (previously  quoted)  was  repeated 

verbatim.  There  were  no  substantive  changes  to  the  document  that  would 
Indicate  a shift  in  T&E  policy  from  the  emphasis  on  early  DT&E  and  lOT&E. 


Personal  observations  of  the  author  based  on  experience  at  the  working 
level  in  1974-1975. 
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SECTION  III 


CURRENT  OT&E  POLICY 
POD  Directive  5000.1 

Major  changes  occurred  approximately  one  year  later  when  the 
new  5000.1  titled  "Major  Systems  Acquisition"  was  issued  on  18  January 
1977.  Its  purpose  was  to  Implement  0MB  Circular  A-109,  also  titled 
"Major  Systems  Acquisition",  dated  April  5,  1976,  and  to  update 
POD  policy  on  acquisition  management.  Considerable  discussion  on 
the  role  of  the  program  manager  is  included.  It  serves  to  further 
emphasize  his  overall  responsibility  for  the  development  of  the  system. 

The  T&E  paragraph  had  two  changes.  Without  defining  the  terms,  the 
Directive  added  the  requirement  for  "An  estimate  of  military  utility 
and  of  operational  effectiveness..."  to  those  of  operational  suitability 
and  logistic  support  requirements.  (14:8)  Significantly,  the  new  T&E 
paragraph  deleted  the  requirement  that  the  operational  test  results  be 
presented  to  the  DSARC  at  the  time  of  production  decision.  The  former 
change  appears  to  be  a direct  quote  from  the  January  1973  DODD  5000.3 
and  is  not  significant.  The  latter  was  clarified  in  a subsequent 
interview  with  DDR&E  (T&E)  officials.  The  change  reflects  the  decentrali- 
zation trend  called  for  in  0MB  Circular  A-109.  The  lOT&E  results  are 
now  to  be  reported  to  the  Service  SARC  (AFSARC)  which  is  to  be 
chaired  by  the  Service  Secretary  or  Assistant  Secretary.  He  in  turn 
will  report  the  combined  DT&E/OT&E  results  to  the  DSARC  as  appropriate. 

POD  Directive  5000.3 

While  the  new  5000.1  has  little  impact  on  T&E,  the  subsequent 
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revision  of  5000.3  contains  major  policy  changes.  Some  of  these  changes 
appear  to  stem  from  the  April  2,  1974  Defense  Science  Board  (DSB) "Report 
of  Task  Force  on  Test  & Evaluation".  (The  reader  is  encouraged  to  obtain 
a copy  of  and  review  the  DSB  report  as  it  contains  valuable  guidelines  on 
T&E.)  In  testimony  before  the  R&D  Subcommittee  of  the  House  Armed  Services 
Committee  on  16  March  1977,  General  Lotz^  stated  that  "...we  are  making 
changes  in  our  test  and  evaluation  policies  and  directives  to  emphasize 
the  need  for  conducting  operational  testing  earlier  in  the  development 
cycle  of  a weapons  sytem."  He  went  on  to  state  four  reasons  for  the  need 
for  earlier  OT&E: 

a.  In  considering  system  concepts  during  the  early  stages 

of  a program,  we  need  to  know  not  only  whether  they  are  technically 
feasible  but  also  whether  they  are  operationally  viable  or 
tactically  sound. 

b.  In  order  to  have  any  impact  on  the  research  and  develop- 
ment phase  of  a program,  OT&E  should  be  conducted  much  earlier 
than  at  DSARC  III  since  by  then  most  R&D  resources  will  have  been 
expended . 

c.  The  earlier  in  the  development  cycle  a deficiency  is 
identified  and  corrected,  the  less  the  cost  of  that  action. 

d.  A trend  toward  initiating  limited  production  before 

the  major  production  decision  results  in  a commitment  of  procure- 
ment funds  prior  to  the  major  production  decision.  (20:9-10) 

An  analysis  of  the  7 March  1977  draft  revision  to  DODD  5000.3  revealed 

the  following  significant  changes  relative  to  planning,  timing  and 

reporting  of  OT&E.  In  general,  the  directive  requires  that  the  "critical 

issues,  test  criteria  and  measures  of  effectiveness  related  to  the 

satisfaction  of  mission  need  shall  be  established  prior  to  the  commencement 

of  tests."  (16:2)  Further,  it  requires  the  establishment  of  a management 

^Lt  General  Walter  E.  Lotz,  Jr.,  USA  (Ret)  Deputy  Director  (T&E)  Office 
of  the  Director  of  Defense  Research  and  Engineering  (DDR&E) . 
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reserve  in  both  schedule  and  funds  to  cover  contingency  testing  in  case 
significant  test  objectives  have  not  been  met.  (16:3)  The  term 
"management  reserve"  is  not  used  but  the  intent  is  clear. 

More  specifically,  OT&E  is  directly  islated  to  the  modified  milestone 
concept  and  the  requirements  to  assess  the  system's  vulnerability  and 
it  capability  against  enemy  countermeasures  are  added.  The  1973  policy 
did  not  call  for  OT&E  until  in  the  Full-Scale  Development  Phase.  The 
1977  policy  requires  lOT&E  to  begin  in  the  Conceptual  Phase  following 
Milestone  0,  to  assess  the  operational  impact  of  the  alternative  technical 
approaches.  lOT&E  will  continue  in  the  Demonstration  and  Validation  Phase 
"to  provide  information  relative  to  projected  operational  effectiveness 
and  suitability  of  the  candidate  system  design  concepts.  (16:6) 
Significantly,  these  lOT&E  results  may  (as  opposed  to  will)  be  considered 
at  Milestone  II  and  for  any  commitment  of  funds  for  long  lead  items  or 
limited  production.  The  previously  specified  Full-Scale  Engineering 
Development  Phase  involvement  remains  the  same  except  that  the  lOT&E  report 
to  support  the  production  decision  will  be  made  to  the  AFSARC  instead 
of  the  DSARC. 

AFTEC  is  now  directed  to  participate  in  DT&E  planning  to  determine 
what  portion  of  DT&E  will  contribute  to  the  accomplishment  of  OT&E 
objectives.  This  early  coordination  should  serve  to  reduce  duplication 
of  effort,  minimize  required  test  resources  and  provide  maximum  data 
that  will  satisfy  common  needs  of  the  developer  (AFSC)  and  AFTEC. 

A subtle  wording  change  relative  to  combined  DT&E  and  lOT&E  testing 
has  put  a more  realistic  emphasis  on  the  issue  in  these  times  of  tight 
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I money.  The  1973  directive  stated  that  "Operational  testing  should  be 

I separate  from  development  testing.  However .... (they) .. .may  be  combined 

E 

I where  separation  would  cause  delay,... or  would ... increase  the  acquisition 

! cost  of  the  system."  (15:4)  The  new  directive  modifies  the  statement 

i to  say  "Development  testing  and  operational  testing  may  be  combined. . 

..".  (16:8)  The  remainder  of  the  words  are  essentially  the  same  but  the 
emphasis  is  shifted  to  combined  testing  by  stating  it  first.  Interviews 
with  DDR&E  officials  confirmed  the  intent  of  the  change.  However,  a 
sentence  added  at  Navy  insistence  clouds  the  issue  slightly:  "As  a 

normal  practice  the  operational  tests  supporting  a production  decision 
will  be  conducted  independently  by  the  OT&E  agency."  (16:9)  Despite 
this  add-on,  the  availability  of  resources  will  always  be  the  final 
determinant.  But  the  emphasis  is  definitely  shifting  toward  the  combined 
testing  concept  as  being  the  most  efficient  and  cost  effective  method 
of  T&E.  The  key  is  to  ensure  that  the  combined  tests  are  planned  and 
executed  to  realistically  provide  the  necessary  test  information;  that 
AFTEC  participates  actively;  and  that  AFTEC  conducts  a separate  evaluation 
of  the  test  data. 

The  1974  DSB  report  stated  that  lOT&E  "tests  should  not  be  conducted 
until  the  primary  objectives  of  the  DT&E  have  been  met."  (5:4)  In  an 
apparent  partial  response,  the  new  5000.3  calls  for  AFTEC  to  "Monitor 
and  review  the  results  of  DT&E... to  assess  the  readiness  of  the  system  for 
operational  testing."  (16:7)  Close,  early  Involvement  of  AFTEC  in  the 
acquisition  process  should  reduce  the  inevitable  debate  between  the 
engineer  who  is  never  finished  and  the  operational  tester  who  is  anxious 
to  start. 
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There  is  an  increased  emphasis  placed  on  the  timely  preparation  of 
the  T&E  Master  Plan  (TEMP) . Up  to  now  it  had  to  be  prepared  "prior  to 
initiation  of  Full-Scale  Development"  and  was  to  "integrate  the  effort 
and  schedules".  Now  it  must  be  prepared  not  later  than  120  days  after 
Milestone  I and  "should  Identify  and  integrate  objectives,  responsibilities, 
resources,  and  schedules  for  all  T&E  to  be  accomplished..."  (16:13)  This 
requirement  alone  would  result  in  earlier  involvement  of  AFTEC  because 
without  its  participation  the  TEMP  would  be  incomplete.  The  same  em- 
phasis is  put  on  the  test  portion  of  the  DCP  (now  called  Decision  Coord- 
inating Paper).  AFTEC  inputs  will  be  required  to  prepare  a complete  DCP. 

There  are  many  other  changes  in  overall  T&E  policy  in  the  1977 
version  of  5000.3.  They  are,  however,  beyond  the  scope  of  this  paper 
and  are  left  to  the  further  investigation  of  the  reader  as  required. 

AF  Regulations 

Air  Force  policy  on  T&E  has  not  had  time  to  react  to  the  new  DOD 
policies.  The  AF  T&E  policy  regulation,  AFR  80-14,  and  the  AFTEC 
regulation,  AFR  23-36,  are  both  mid-1976  documents.  The  most  current 
element  is  the  AFSC  Supplement  to  80-14,  dated  3 January  1977. 

Two  key  elements  of  the  current  80-14  are  the  relationship  between 
the  program  manager  and  AFTEC  and  the  conduct  of  combined  testing. 

"The  program  manager  has  overall  responsibility  for  a system  acquisition 
program  (except  the  management  of  OT&E)."  (1:3)  He  will  include  OT&E 
requirements  in  the  test  program  and  support  OT&E  as  appropriate.  "AFTEC 
has  responsibility  for  managing  the  OT&E  in  a major  acquisition  program... 
AFTEC  will  plan,  direct,  conduct,  control,  and  independently  evaluate 
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and  report  on  OT&E."  (1:3)  This  potential  conflict  of  roles  will  be 
covered  further  in  the  interview  section  of  this  paper. 

Paragraph  17  of  AFR  80-14,  "Conducting  a Combined  Test  Program" 
seems  to  negate  any  sort  of  team  concept  in  combined  testing.  The 
planning  aspects  of  the  paragraph  conflict  with  the  nev/  5000.3  in  that 
they  call  for  separate  DT&E  and  OT&E  test  plans  with  the  program 
manager  responsible  for  integrating  the  OT&E  plan.  Joint  planning  is 
not  mentioned.  The  separation  of  roles  and  functions  continues  down 
to  the  "joint"  test  force  (JET).  AFSC  appoints  the  test  force  director 
who  is  responsible  for  DT&E,  Integrating  combined  test  events,  insuring 
availability  of  resources  and  insuring  the  safety  of  the  test  program. 
AFTEC,  on  the  other  hand,  will  provide  the  OT&E  test  director  who 
manages  the  OT&E  portion  of  the  combined  program.  Previously  this 
function  was  performed  by  the  user  detachment  commander  serving  as 
Deputy  JTF  Director  as  opposed  to  "OT&E  test  director."  Thus  there 
is  one  JTF  with  two  directors  - a situation  that  is  difficult  to  depict 
on  an  organizational  chart  and  which  tends  to  splinter  the  "test  team". 

AFSC  Supplement  1 to  80-14  attempts  to  improve  the  situation  by  stating 
that  "This  test  force  must  be  able  to  function  efficiently  as  an  entity 
to  accomplish  overall  program  test  objectives;  management  relations 
must  be  clearly  defined".  (Underlining  added  for  emphasis)  (1:3) 
Unfortunately  neither  80-14  nor  AFSC  Supp  1 specifies  one  individual 
who  has  overall  test  program  responsibility.  AFR  23-36  generally  follows 
the  guidance  set  forth  in  AFR  80-14.  Appropriate  Air  Staff  agencies 
are  aware  of  the  difficulties  with  the  combined  test  program  guidance 
and  hopefully  will  provide  improved  guidance  in  the  next  revision  of  80-14. 
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AFTEC  has  been  in  existence  for  more  than  three  years.  Initially 
there  were  serious  growing  pains  as  this  new  organization  was  imposed 
upon  on-going  programs.  An  adversary  had  been  thrown  into  a predominantly 
advocate  environment.  Very  few  individuals  really  understood  the  role 
of  the  independent  test  agency.  The  situation  has  now  had  time  to 
settle  down  and  AFTEC  is  fully  operational.  In  fact,  sixty  additional 
personnel  were  recently  authorized.  The  AFTEC  FY1978  budget  calls  for 
$17.9  million  in  3080  (Other  Procurement),  3400  (O&M)  and  3600  (RDT&E) 
money.  This  is  more  than  double  the  $8.46  million  FY1976  budget  and 
the  size  of  the  budget  is  expected  to  continue  to  rlse.^  No  figures 
were  available  on  the  cost  of  AFTEC  managed  OT&E  but  it  can  be  assumed 
to  be  significant  since  approximately  15%  of  the  annual  DOD  RDT&E  cost 
for  weapons  systems  is  used  for  test  and  evaluation.  (20:2) 


Telecon  with  Mr.  H.  Jensen,  AFTEC,  Plans  and  Resources,  Kirtland 
AFB,  N.M. 
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SECTION  IV 
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COMPARATIVE  VIEWS  ON  OT&E 

With  the  relative  significance  of  AFTEC  and  OT&E  established,  a 
series  of  structured  interviews  were  conducted  to  determine  the 
current  view  of  OT&E  in  the  Air  Force.  These  interviews,  through  a 
series  of  nine  questions,  compare  the  views  of  various  DOD  and  AF 
officials  with  those  of  the  AFTEC  Commander.  Officials  from  both  DT&E 
and  OT&E  organizations  were  interviewed  as  well  as  senior  DD(T&E) 
officials . 

To  determine  the ■ reason  behind  reissuance  of  DODD  5000.3,  a 
separate  question  was  asked  of  Admiral  Kollmorgen.  "Why  was  a new 
5000.3  required  at  this  time?"  The  primary  reason  was  to  get  OT&E 
started  earlier  in  an  attempt  to  establish  the  operational  environment 
while  the  design  engineers  were  still  working  the  design  on  paper. 

In  this  way  there  would  be  better  assurance  that  the  design  would  i 

consider  the  operational  environment  and  not  just  the  performance  ' 

specification.  A second  reason  was  to  encourage  combined  DT&E  and 
lOT&E  testing  and  common  use  of  the  test  resources  and  test  results.  (25) 

The  draft  5000.3  seems  to  be  following  this  intended  path. 

The  approach  used  to  analyze  the  interviews  was  to  determine  if 
a consensus  of  opinion  existed  on  the  issues  and  then  compare  the 
results  with  AFTEC 's  view.  It  should  be  noted  that  the  interviewees  were 
expressing  their  personal  opinion  and  not  necessarily  Air  Force  or  DOD 
policy. 

1.  DODD  5000.1  calls  for  a "strong  SPO  to  achieve 
program  objectives".  AFR  80-1A  says  that  the  PM 
has  "overall  responsibility"  except  for  OT&E.  Does 
this  in  effect  dilute  the  PM's  authority? 

u 
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with  minor  reservations,  the  consensus  was  that  there  is  no  significant 
problem  with  this  concept  at  this  time.  The  view  was  that  since  the 
PM  has  control  of  the  resources  of  the  program,  dollars  and  the  test 
articles,  he  is  in  a strong  negotiating  position  when  the  original  test 
plans  are  being  developed.  His  task  is  a delicate  one  that  requires 
diplomacy  and  tact  in  establishing  effective  working  relationships  with 
AFTEC.  The  same  reasoning  applies  at  the  JTF  level  where  the  split 
roles  are  also  evident.  The  JTF  Director  has  the  necessary  authoritv 
he  needs  through  his  "ownership"  of  the  test  articles  and  responsibility 
for  safety  of  operations.  It  is  important  to  maintain  the  concept  of 
independence  but  not  at  overall  program  expense.  In  the  final  analysis, 
as  Admiral  Kollmorgen  stated,  the  golden  rule  applies:  "He  who  has  the 
gold,  rules."  (25)  Thertln  lies  the  authority  that  the  PM  needs  and 
that  he  can  use  as  appropriate. 

The  AFTEC  Command,  Major  General  Leaf,  views  this  issue  in  a 
similar  fashion.  Rather  than  viewing  the  PM's  authority  as  being 
diluted,  he  sees  the  situation  as  merely  an  affirmation  of  the  differing 
roles  of  DT&E  and  lOT&E.  The  former  being  specification  oriented  and 
the  latter  being  oriented  toward  the  operational  environment.  Further- 
more, because  of  the  complementary  nature  of  DT&E  and  lOT&E,  the  PM's 
ability  to  produce  an  operationally  effective  and  suitable  system  is 
enhanced . 

As  AFTEC  has  matured  and  the  rest  of  the  AF  has  gained  a better 
understanding  of  its  role,  significant  conflicts  in  this  area  have 
diminished.  However,  as  Mr.  Boykin  stated,  the  issue  should  be  put  into 
clearer  context  in  a subsequent  change  to  DODD  5000.1,  the  basic 
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acquisition  management  directive,  and  AFR  80-14.  Further  definitioi; 
would  preclude  problems  from  reoccuring  as  personalities  Involved  change. 

2.  How  much  should  lOT&E  requirements  influence  total 
Full  Scale  Engineering  Development  test  schedules  and 
the  Milestone  III  date? 

There  was  no  clearcut  answer  to  this  question.  Since  all  programs 
are  different,  the  effect  will  be  different.  The  common  thread  through- 
out the  responses  was  that  well  planned,  achievement  oriented  program 
(vice  time-schedule  oriented)  would  have  no  problems  with  the  DSARC 
process.  There  is  a danger  of  unnecessary  program  stretchout  if  the 
test  program  is  not  well  structured  since  the  requirement  for  an  lOT&E 
input  to  the  production  decision  process  is  a firm  one.  Both  the  PM 
and  AFTEC  must  realize  that  lOT&E  is  not  an  end  Itself  but  rather  it  is 
one  of  the  means  to  the  end  objective  of  selecting  the  correct  production 
decision.  When  the  areas  of  risk  have  been  adequately  addressed,  go 
for  production,  don't  continue  to  test  to  obsolescence. 

Again  AFTEC  concurred  by  emphasizing  that  there  should  be  no  adverse 
Influence  if  the  lOT&E  requirements  are  integrated  into  the  program  early 
and  are  properly  reflected  in  the  Statement  of  Work  and  the  contract. 

3.  Have  we  swung  from  too  little  to  too  much  testing 
(DT&E  or  lOT&E)  prior  to  Milestone  III? 

There  was  wide  disagreement  on  this  one.  Interestingly,  the 
developer  oriented  responses  indicated  that  we  were  doing  too  much 
testing  or  at  least  leaning  that  way.  There  was  also  a feeling  that, 
regardless  of  amount,  the  areas  of  emphasis  need  to  be  scrutinized 
closely.  There  was  a concern  that  overemphasis  on  lOT&E  at  the  expense 
of  DT&E  was  dangerous  and  that  developmental  "show  stoppers"  could  be 
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missed.  The  best  summation  in  this  area  is  that  the  system  should  be 
developed  first  and  the  concept  of  combined  developmental  and  operational 
testing  (with  independent  evaluation)  be  used  to  the  maximum  extent 
practicable. 

The  AFTEC  response  quoted  the  1977  Defense  Science  Board  report: 
"There  appears  to  be  little  or  no  overtesting  done  under  the  DOD 
directives;  what  testing  is  done  contributes  its  full  value  to  the 
improvement  and  verification  of  system  performance."  (6:4) 

4.  Should  the  cost  of  lOT&E  be  borne  by  the  SPO? 

Wouldn't  putting  it  in  the  AFTEC  budget  provide 
truer  figures  on  the  actual  cost  of  OT&E? 

This  issue  was  not  considered  a significant  problem  by  the  majority 
of  interviewees.  All  but  one  felt  that  the  system  should  remain  as  is 
since  it  preserves  the  PM's  negotiation  position  through  his  control 
of  the  resources  required  for  the  lOT&E  program.  Furthermore,  if  all 
funding  control  were  transferred  to  AFTEC  there  is  a real  danger  of 
creating  an  "operational  monster".  The  present  system  allows  for 
checks  and  balances. 

AFTEC  has  picked  up  some  of  the  lOT&E  costs  previously  funded  by 
the  PM.  These  are  command  and  special  support  lOT&E  costs.  General 
T eaf  sees  no  need  for  his  organization  to  assume  any  of  the  additional 
costs  of  lOT&E  and  agrees  that  it  is  not  a significant  problem. 

5.  What  are  we  getting  now  in  terms  of  operational 
assessment  that  was  not  available  prior  to  1974  when 
AFSC  and  the  users  were  working  together?  In  other 
words,  what  new  contribution  is  AFTEC  making? 

6.  The  AFTEC  budget  projection  for  FY  1978  is  $17.9  mil. 
and  will  be  higher  in  the  outyears.  Do  you  feel  that 

we  are  getting  our  money's  worth? 
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The  only  apparent  consensus  on  these  two  questions  was  that 
AFTEC's  main  contribution  was  its  independent  test  report  to  the  Chief 
of  Staff  and  the  AFSARC.  Otherwise,  the  responses  were  mixed.  Some 
felt  that  AFTEC's  contributions  were  very  worthwhile  but  that  their 
cost  effectiveness  had  not  yet  been  proven.  Others  felt  we  were  doing 
just  as  well  prior  to  AFTEC's  involvement.  Whether  or  not  AFTEC's 
contribution  is  cost  effective  will  not  be  known  until  someone  can 
determine  a way  of  measuring  costs  avoided  due  to  AFTEC's  input.  Any 
deficiency  found  early  enough  to  correct  during  production  instead  of 
through  retrofit  equates  to  costs  avoided.  In  the  future,  it  may  be 
possible  to  develop  a means  to  compare  the  number  of  post-production 
changes  before  AFTEC  (1974)  and  after  AFTEC.  Only  then  can  some 
statistically  valid  measure  of  cost  effectiveness  be  made.  But  until 
then,  the  AFTEC  contribution  is  recognized  as  significant  in  the  AF 
and  at  DOD  and  Congressional  levels. 

The  AFTEC  response  reinforced  the  positive  aspects  of  OT&E  in  the 
acquisition  process  by  citing  several  specific  contributions.  In  addition 
to  the  reporting  aspect,  the  contributions  include: 

a)  revision  of  the  deficiency  reporting  process  and  providing 
better  visibility,  prioritization  of  deficiencies,  and  Identification  of 
potential  fixes  through  coordination  with  the  developer  and  the  user; 

b)  standardization  of  OT&E  planning,  criteria,  execution  and 

f support  as  well  as  evaluation  methodology; 

I c)  improved  software  evaluation  through  the  use  of  a software 

r 

evaluation  group; 
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d)  earlier  involvement  of  OT&E  personnel  in  program  at  PM  and 


I 


contractor  facilities;  (It  should  be  noted  that  user  personnel  were 
co-located  at  both  PM  and  contractor  facilities  on  the  B-1  program 
prior  to  AFTEC's  inception  in  1974.) 

e)  improved  logistics  and  manpower  assessments  and  operations 
and  support  cost  estimates. 

From  a cost  effectiveness  (cost  avoidance)  standpoint,  the  early 
identification  of  problems  when  they  are  still  relatively  cheap  to 
fix  is  most  significant.  The  example  of  the  F-15  production  radar  is 
a case  in  point.  Major  General  Leaf  pointed  out  that  the  APG-63  prod- 
uction radar  would  not  have  been  fixed  without  AFTEC  having  surfaced 
the  problem  at  high  levels  in  the  AF. 

7.  AFTEC  has  been  criticized  as  a 'nay  sayer'.  Is 
this  a valid  criticism? 

Initially,  the  AFTEC  reports  seemed  excessively  negative  and  with- 
out balance.  The  interviewees  generally  agreed  that  this  had  been  true 
but  that  it  was  no  longer  a significant  factor.  They  felt  that  AFTEC 
personnel  have  matured  in  the  job  and  that  they,  as  well  as  the  rest  of 
the  AF,  have  a better  understanding  of  their  role.  They  agreed  that 
AFTl  should  be  objective  - neither  success  nor  failure  oriented  - and 
should  present  a balanced  report.  There  remains  the  danger  of  the  lOT&E 
comments  being  taken  out  of  context  outside  of  the  AF.  This  latter 
danger  makes  the  need  for  balanced  objectivity  even  more  important. 

The  AFTEC  response  agrees  with  the  above.  The  AFTEC  charter  calls 
for  evaluation  without  bias  or  prejudice.  The  objective  is  to  provide 
an  operational  assessment  that  highlights  both  the  positive  and 
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negative  aspects  of  the  system  being  evaluated. 

8.  Does  it  really  matter  who  gathers  the  flight  test 
data  (developer  or  operator)  if  the  test  conditions 
are  operationally  representative? 

This  has  been  an  emotional  issue.  In  some  programs,  such  as  the 
B-1  and  F-16,  the  "negotiations"  reached  the  general  officer  level  to 
determine  the  required  mix.  In  an  unlimited  resource  environment  there 
would  be  no  issue  since  lOT&E  and  DT&E  can  be  clearly  separated.  In 
combined  programs  with  limited  resources  it  is  significant.  The  DDR&E 
(T&E)  view  is  that  you  "can't  have  an  operational  test  if  it  is  all 
done  by  developers."  (25)  Others  felt  that  a data  point  was  a data 
point.  But  overall,  there  was  agreement  that  the  operational  pilot  has 
a different  perspective  than  that  of  the  trained  test  pilot  who  has  been 
out  of  the  operational  environment  for  a number  of  years.  Qualitative 
assessments  by  both  developer  and  user  pilots  are  required  for  a bal- 
anced evaluation.  Another  aspect  was  colorfully  tagged  the  "hamburger" 
factor.  (23)  Pilot  skills  vary.  If  only  test  pilots  and  the  best 
operational  pilots  evaluate  the  new  aircraft,  they  might  unconsciously 
compensate  for  a characteristic  that  a pilot  of  lesser  skill  would  find 
unacceptable.  Thus  there  is  a need  for  a mix  of  pilots.  The  PM  and 
AFTEC  must  jointly  agree  on  the  proper  mix. 

From  the  standpoint  of  pure  performance  and  flying  qualities  data 
taking,  AFTEC  sees  no  requirement  to  specify  who  flies  the  points  since 
they  are  quantitative  in  nature.  But  this  is  not  true  for  the  operational 
effectiveness  evaluation.  For  lOT&E,  the  user  input  is  mandatory.  Thus 
AFTEC  concurs  with  the  overall  consensus  of  opinion. 
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9.  AFTEC  currently  is  limited  to  "managing"  the 
lOT&E  program.  Should  it  also  have  the  capability 
to  "conduct"  lOT&E  as  does  its  Navy  counterpart? 


The  unanimous  response  to  this  final  question  was  "no".  Such 
a concept  would  inevitably  lead  to  another  "super  organization"  or 
"operational  monster"  that  the  Air  Force  cannot  afford,  that  would 
be  redundant,  and  that  would  absorb  a lot  of  the  MAJCOM  efforts  that 
rightly  belong  to  them  for  follow-on  OT&E.  Further  studies  may  come 
up  with  a better  system  than  currently  exists  but  the  current  concept 
is  considered  much  better  for  the  AF  than  the  Navy  approach  of  a 
separate  operational  test  and  evaluation  force. 

Not  surprisingly,  the  AFTEC  response  was  in  complete  agreement 
with  the  unanimous  response  noted  above.  It  pointed  out  that  the 
current  concept  is  in  concert  with  the  new  5000.3.  Since  the  AFTEC 
test  team  personnel  are  "borrowed"  from  the  using  commands  they  are 
more  current  in  operational  matters  than  they  would  be  if  they  were 
assigned  to  an  independent  agency.  There  is,  as  a result,  less  danger 
of  their  becoming  "professional  testers"  and  losing  sight  of  their 
role.  Furthermore,  the  sharing  of  test  resources  fosters  combined 
DT&E/IOT&E  testing  which  is  Inherently  more  cost  effective  if  done 
properly. 
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SECTION  V 


INCORPORATING  OT&E  INTO  THE  PROGRAM 
Regardless  of  the  variety  of  opinion  on  the  utility  and  •»f fectiveness 
of  AFTEC  in  the  acquisition  process,  the  fact  remains  that  AFTEC  is  here. 
The  AFTEC  contribution  is  highly  visible  and  time  sensitive  in  the  decision 
process.  A senior  official  stated  that  "If  you  have  good  people  trying 
to  do  a good  job,  it  comes  out  right".  This  may  be  true,  but  in  the  final 
analysis  it  is  the  program  manager  that  is  held  responsible  for  the 
program  outcome.  How  effectively  the  valid  operational  requirements  are 
incorporated  into  the  program  will  be  a major  determinent  of  the  program 
outcome.  What  can  the  program  manager  do  to  ensure  that  all  valid 
operational  requirements  are  adequately  covered  in  his  program? 

The  answer  to  this  extremely  important  question  can  be  broken  down 
into  three  major  areas: 

1)  Establishing  the  baseline 

2)  Organizing 

3)  Planning 

Establishing  the  Baseline 

Establishing  the  baseline  involves  a number  of  elements.  First  of 
all,  know  the  significant  DOD  Directives  and  AF  Regulations.  These  in- 
clude DODD  5000.1,  5000.2,  and  5000.3.  DODD  5000.1  for  overall  guidance; 
DODD  5000.2  for  the  Milestone  checklists  and  DCP  content;  and  DODD  5000.3 
for  the  specific  DOD  T&E  policy.  AFR  80-14  and  AFSC  Supp  1 are  also  key. 
AFSCP  800-3,  "A  Guide  to  Program  Management",  is  also  an  excellent 
reference.  Second,  know  the  operational  concept  for  the  system.  This 
is  the  key  baseline.  It  has  to  be  included  in  the  system  design  as  well 
as  the  test  plan.  It  will  serve  as  the  criteria  against  which  the  system 
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effectiveness  is  measured.  A new  AF  policy  for  initiating,  developing. 


updating  and  approving  operational  concepts  for  new  and  improved  major 
systems  is  being  considered  and  the  PM  has  to  be  involved  if  the 
system  he  is  developing  is  to  meet  the  need.  Figure  1 depicts  the 
flow  of  the  operational  concept.  (29:12B)  The  Required  Operational 
Capability  (ROC)  has  always  been  part  of  the  system  documentation.  It 
leads  to  the  Mission  Element  Need  Statement  (MENS)  which  is  a new 
requirement  under  DODD  5000.1.  The  PM  needs  to  be  intimately  familiar 
with  both  the  ROC  and  the  MENS.  The  proposed  operational  concept  will 
be  a dynamic  document  tied  to  program  milestones  for  update  and  refinement. 

AFSC  supports  this  concept  and  the  requirement  for  approval  of  operational 
concept  changes  to  preclude  inadvertent  breaches  of  program  thresholds. 

The  concept  format  contains  operational  thresholds,  forces,  standards  for 
development,  organization,  basing  and  support,  and  significantly,  the 
updated  baseline  for  system  design  and  T&E.  The  benefits  for  the  PM 
from  this  concept  are;  that  it  will  provide  a forum  for  improved  dialogue 
between  his  organization,  the  user,  supporter  and  AFTEC;  it  will  improve 
means  to  ensure  that  the  system  design  reflects  the  operational  concept; 
and  ensures  corporate  reviews  for  changes.  In  the  words  of  Lt  Colonel 
Delligatti:  "The  corporate  inputs  to  the  initial  and  subsequent  updates 

of  the  operational  concept  provides  the  user,  developer,  supporter  and 
tester  a voice  in  the  operational  concept.  With  this  approach,  the  system 
design  will  reflect  the  operational  concept.  At  initiation  and  each 
update,  the  operational  concept  document  is  required  to  go  to  all  applicable 
agencies  for  review  and  comment  prior  to  being  submitted  by  the  user  to 
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AF/XO  for  Headquarters  USAF  approval.  The  proposed  operational  concept 
directive  will  provide  a means  to  ensure  that  the  operational  community 
complements  the  system  acquisition  process  and  help  to  ensure  that  we 
get  the  best  system  for  the  money."  (29:12B)  Whether  or  not  the  above 
system  becomes  official  AF  policy,  it  is  the  type  of  approach  that  the 
PM  should  use  the  ensure  that  all  parties  concerned  with  his  system  are 
working  from  the  same  baseline. 

Third,  establish  the  critical  questions  and  issues  to  be  addressed 
in  your  test  program.  This  will  require  coordination  with  such  organ- 
izations as  DDR&E(T&E) , AF/RD  and  XO,  AFSC  and  AFTEC/Using  Command. 

These  will  be  the  questions  that  the  PM  will  have  to  address  at  the 
program  Milestones  to  gain  approval  to  proceed  to  the  next  phase  of  the 
development  cycle.  They  will  be  reflected  in  the  DCP  (refer  to  DODD 
5000.2),  the  Test  and  Evaluation  Objectives  Annex  (TEOA)  of  the  Program 
Management  Directive  (refer  to  AFR  80-14)  , the  Test  and  Evaluation 
Master  Plan  (TEMP)  (refer  to  AFR  80-14/AFSC  Sup  1)  and  the  PM's  own 
Program  Management  Plan  (PMP)  (refer  to  AFSCP  800-3) . 

Organizing 

The  PM  cannot  accomplish  all  of  the  above  tasks  without  the  assist- 
ance of  a strong  organization  composed  of  the  right  people.  The  key  man 
will  be  his  Director  of  Test  and  Evaluation.  For  an  aircraft  program, 
this  individual  should  be  a qualified  test  pilot  who  is  current  in  sim- 
ilar type  aircraft.  It  should  be  a rated  position  and  require  attendance 
at  the  Executive  Refresher  Course  at  the  Defense  Systems  Management 
College.  The  test  office  should  be  sufficiently  manned  to  be  the  single 
voice  of  test  for  the  PM  and  should  be  responsible  for  all  ground  and 
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flight  tests  throughout  the  program  to  preclude  conflicting  guidance 
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to  the  contractor. 

Planning 

With  the  test  office  as  the  core,  the  PM  should  form  the  Test  Planning 
Working  Group  (TPWG)  in  accordance  with  APR  80-14/AFSC  Sup  1.  This  group 
should  include  all  agencies  that  will  be  Involved  in  the  test  program; 
the  responsible  test  organization  (RTO) , AFTEC,  the  using  and  supporting 
commands,  and  the  system  contractor  if  already  Identified.  The  TPWG 
concept  worked  well  in  the  past  before  it  was  required  by  80-14.  It 
was  used  extensively  and  very  successfully  in  the  B-1  test  program.  It 
provided  the  necessary  forum  for  all  test  related  subjects  ranging  from 
establishing  objectives  and  baselines  to  defining  organizational 
responsibilities.  With  the  combined  inputs  of  all  organizations  along 
with  the  operational  concept  and  critical  Issues,  the  TPWG  can  prepare 
the  draft  TEMP,  TEOA,  and  the  test  portion  of  the  PMP  and  DCP.  It  will 
also  be  involved  in  the  preparation  of  the  detailed  test  plans  whether 
or  not  the  contractor  is  assigned  primary  planning  responsibility. 

The  PM  has  to  be  directly  Involved  in  this  planning  process.  Based 
on  his  program  charter,  he  must  provide  specific  guidance  to  his  Director 
of  Test.  While  the  PM  is  responsible  for  integrating  the  lOT&E 
requirements  into  his  program,  he  should  not  blindly  accept  them.  He 

must  question  all  requirements  and  changes  to  requirements  to  verify  j 

their  validity  and  need.  He  should  ensure  that  the  requirements  are 

listed  and  prioritized  as  specific  objectives  rather  than  as  percentages 

of  flying  hours  or  missions.  He  must  ensure  that  the  aircraft  is 

adequately  developed  through  the  accomplishment  of  prioritized  DT&E 
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objectives  before  jointly  agreeing  with  AFTEC  that  the  system  is 


ready  for  lOT&E.  Finally,  he  must  ensure  that  the  output  of  the  planning 
process  is  a jointly  developed  strategy  that  is  agreed  to  by  all  parties 
concerned.  It  is  only  through  a coordinated,  program  oriented  plan 
that  the  PM  can  ensure  that  he  will  not  be  surprised  later  in  the  program 
by  a previously  unidentified  requirement  that  is  either  out  of  scope 
or  severely  impacts  his  schedule  or  both.  The  earlier  in  the  program 
that  he  can  gain  this  confidence  the  better  off  both  he  and  his  program 
will  be.  Finally,  he  will  have  complied  with  both  the  spirit  and  the 
intent  of  DODD  5000.3  through  his  early  incorporation  of  the  lOT&E 
requirements . 
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SECTION  VI 


SUMMARY 

In  summary,  this  paper  has  looked  at  the  criticisms  and  changes 
in  DOD  and  AF  policy  on  operational  test  and  evaluation  from  1970  through 
early  1977.  It  analyzed  the  new  5000.1  and  the  soon  to  be  released  draft 
5000.3.  Through  a series  of  interviews,  it  compared  the  views  of  various 
DOD  and  AF  elements  on  the  effectiveness  of  AFTEC  with  AFTEC's  view  of 
itself.  Finally,  the  policies  and  views  were  combined  into  recommendations 
for  the  PM  to  follow  to  ensure  that  the  valid  operational  requirements 
are  Included  in  his  program.  Each  program  is  different  and  no  universal 
set  of  guidelines  can  be  developed  that  will  apply  to  them  all.  But  the 
aware  PM  will  consider  them  all  and  tailor  the  tactics  to  his  specific 
situation.  In  that  way  he  can  be  as  assured  of  success  as  anyone  can  in 
the  changing  political  and  military  environment  of  today's  world. 
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DRAFT 


March  7,  1977 

NUMBER  5000. 3 


DDR&E 


Department  of  Defense  Directive 


SUBJECT 


Test  and  Evaluation 


Refs:  (a)  DoD  Directive  5000.3,  ’’Test  and  Evaluation”, 

January  19,  1973  (hereby  cancelled) 

(b)  DoD  Directive  5000.1,  "Acquisition  of  Defense  Materiel", 
January  18,  1977 

(c)  DoD  Directive  6050.1,  "Environmental  Considerations  in 
DoD  Actions",  March  19,  1974 

(d)  DoD  Directive  4650.1,  "Management  and  Use  of  Radio 
Frequency  Spectrum",  December  13,  1974 

(e)  DoD  Directive  5000.2,  "Major  System  Acquisition  Process", 
January  18,  1977 

(f)  DoD  Directive  3200 . 11 ,' "Use,  Management  and  Operation 
of  Department  of  Defense  Major  Ranges  and  Test 
Facilities",  June  18,  1974 


REISSUANCE  AND  PURPOSE 


This  Directive  establishes  policy  for  the  conduct  of  test  and 
evaluation  by  the  Military  Departments  and  Defense  Agencies 
(hereinafter  referred  to  collectively  as  "DoD  Components")  in 
the  acquisition  of  defense  systems  (Sections  II  through  V) . 

In  addition,  it  defines  the  responsibilities  of  the  Deputy 
Director  of  Defense  Research  and  Engineering,  Test  and 
Evaluation  (Deputy  DDR&E(T&E) ) (Section  VI).  Reference  (a) 
is  hereby  superseded  and  cancelled. 
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II.  SCOPE  AND  APPLICABILITY 


The  provisions  of  this  Directive  encompass  major  defense 
system  acquisition  programs  as  designated  by  the  Secretary 
of  Defense  (described  in  Section  II.  of  reference  (b))  and 
apply  to  all  DoD  Components  that  are  responsible  for  such 
programs.  In  addition,  the  management  of  programs  not 
designated  as  major  system  acquisitions  shall  be  guided  by 
the  principles  set  forth  herein. 

III.  POLICIES  AND  PRINCIPLES 
A.  General . 

1.  Test  and  evaluation  shall  be  commenced  as  early  as 
possible  and  conducted  throughout  the  system  acquisition 
process  as  necessary  to  assess  the  acquisition  risks 
and  to  evaluate  the  operational  effectiveness  and 
operational  suitability  of  the  defense  system  being 
developed.  Meaningful  critical  Issues,  test  criteria 
and  measures  of  effectiveness  related  to  the  satis- 
faction of  mission  need  shall  be  established  prior  to 
the  commencement  of  tests. 

2.  Acquisition  schedules,  financial  plans,  and  contractual 
arrangements  shall  recognize  that  decisions  to  commit 
significant  added  resources  to  a program  shall  be  based, 
inter  alia,  upon  successful  accomplishment  of  test  and 
evaluation  objectives  which  bear  a meaningful  relation- 
ship to  required  operational  capabilities. 
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3.  Schedules  and  funding  commitment  shall  be  established 

to  permit  additional  (contingency)  testing  when  significant 
test  objectives  have  not  been  achieved  during  a planned 
test  or  test  series,  but  a determination  is  made  to  continue 
with  the  program.  Such  additional  testing,  with  positive 
results,  may  be  required  to  support  a decision  to  commit 
significant  additional  resources. 

4.  Before  the  initiation  of  development  of  a new  system, 
test  and  evaluation  using  existing  systems,  or  modi- 
fications thereto,  may  be  appropriate  to  help  define 
the  military  need  for  the  proposed  new  system  and  to 
estimate  its  operational  effectiveness  and  operational 
suitability. 

B,  Development  Test  and  Evaluation  (DT&E) . DT&E  is  that  test 
and  evaluation  conducted  to  assist  the  engineering  design 
and  development  process  and  verify  attainment  of  technical 
performance  specifications  and  objectives.  DT&E  is 
normally  managed  and  monitored  by  the  DoD  Component's 
developing  agency,  which  reports  the  results  to  the 
responsible  Military  Service  Chief  and 
Service  Secretary,  or  Defense  Agency  Director. 

1.  DT&E  includes  testing  of  components,  subsystems, 
related  software,  and  prototype  or  preproduction 
models  of  the  system.  In  addition,  compatibility  and 
interoperability  with  existing  or  planned  equipments 
and  systems  shall  be  tested. 
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2.  During  the  development  phase  following  the  Program 
Initiation  Decision  (Milestone  0)  adequate  T&E  shall 
be  accomplished  when  appropriate  to  select  preferred 
alternative  systems  concepts. 

3.  During  the  development  phase  following  the  Demonstration 
and  Validation  Decision  (Milestone  I) , adequate  DT&E 
shall  be  accomplished  to  identify  the  preferred  technical 
approach  and  to  demonstrate  that  technical  risks  have 
been  identified  and  that  solutions  are  in  hand. 

4.  During  the  Full-Scale  Engineering  Development  Phase 

and  prior  to  the  first  major  production  decision,  the 
DT&E  accomplished  shall  be  adequate  to  insure:  that 

engineering  is  reasonably  complete;  that  all  signifi- 
cant design  problems  (including  vulnerability, 
compatibility,  interoperability,  reliability,  main- 
tainability, and  logistical  considerations)  have  been 
identified;  and  that  solutions  to  these  problems  are 
in  hand. 

5.  During  full-scale  production,  subsequent  to  the 
Milestone  III  decision,  DT&E  will  be  an  integral  part 
of  the  development,  acceptance,  and  introduction  of: 

a.  Product  improvements  into  the  produced  system. 

b.  Operational  characteristic  changes  to  meet  iden- 
tified threat  changes. 


c.  Changes  to  reduce  system  life  cycle  costs. 
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6.  For  those  systems  which  have  an  interface  with  equipment 
;»  of  another  DoD  Component  or  may  be  acquired  by  two  or 

more  Components,  joint  DT&E  may  be  required.  Such  joint 
testing  will  include  participation  and  support  by  all 
affected  Components  as  appropriate. 

' C.  Operational  Test  and  Evaluation  (OT&E) . OT&E  is  that  test 

and  evaluation  conducted  to  estimate  a system’s  operational 
effectiveness  (including  vulnerability)  and  operational  suit- 
ability (including  compatibility,  interoperability,  reliability, 
maintainability,  availability,  logistic  supportability , and 
training  requirements),  as  well  as  the  need  for  any  modifica- 
tions. In  addition,  OT&E  provides  information  on  organization, 
personnel  requirements,  doctrine,  and  tactics.  It  may  also 
provide  data  to  support  or  verify  material  in  operating 
instructions,  publications,  and  handbooks.  OT&E  will  be 
accomplished  by  operational  and  support  personnel  of  the  type 
and  qualifications  of  those  expected  to  use  and  maintain  the 
system  when  deployed,  and  will  be  conducted  in  as  realistic 
an  operational  environment  as  possible,  including  enemy 
countermeasures.  It  will  address  operational  performance 
of  component  systems,  including  new  or  improved  components 
and  their  interaction  with  related  existing  systems.  Costs 
directly  related  to  operational  testing  in  a program  shall 
I be  planned  and  budgeted  as  a part  of  the  total  program  costs. 

I 1.  OT&E  will  normally  be  conducted  in  phases,  each  keyed 

to  an  appropriate  decision  point. 


( 

I 
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a.  During  the  Alternative  System  Exploration  Phase 
following  Milestone  0,  OT&E  will  be  initiated, 
when  appropriate!  to  assess  the  operational  impact 
of  the  candidate  technical  approaches. 

b.  During  the  Demonstration  and  Validation  Phase, 

OT&E  will  be  conducted,  as  necessary,  to  examine 
the  operational  aspect  of  selected  technical 
approaches,  and  to  provide  information  relative 
to  projected  operational  effectiveness  and  suit- 
ability of  the  candidate  system  design  concepts. 

In  addition,  such  OT&E  may  provide  inputs  to 
decisions  made  at  Milestone  II  or  later  to  commit 
funds  for  long  lead  items  or  for  limited  production. 

c.  During  the  Full-Scale  Engineering  Development  Phase, 
following  the  Milestone  II  decision,  adequate  OT&E 
will  be  accomplished  to  provide  a valid  estimate 

of  operational  effectiveness  (including  vulnerability) 
and  operational  suitability  (including  compatibility, 
interoperability,  reliability,  maintainability,  avail- 
ability, logistic  supportability,  and  training  requirei.ients) 

d.  Subsequent  to  the  Milestone  III  decision,  OT&E 
will  be  continued  as  necessary  to  refine  these 
estimates,  to  evaluate  changes,  and  to  re-evaluate 
the  system  to  ascertain  whether  it  continues  to 
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p meet  operational  needs  and  retains  its  effective-  I 

ness  in  a new  environment  or  against  a new  threat. 

2.  In  each  DoD  Component  there  will  be  one  major  field  ■ 

agency,  separate  and  distinct  from  the  materiel 

developing/procuring  command  and  from  the  using  command  I 

i 

which  is  responsible  for  the  conduct  of  OT&E  and  reports  ] 

the  results  of  its  independent  operational  test  and  I 

evaluation  directly  to  the  Military  Service  Chief  and 
Service  Secretary,  or  Defense  Agency  Director.  The 
OT&E  Agency  will; 

a.  Participate  in  planning  of  DT&E  to  ascertain  what 
portion  of  DT&E  will  contribute  to  the  accomplish- 
ment of  OT&E  objectives. 

! 

b.  Monitor  and  review  the  results  of  DT&E  as  necessary 
to  obtain  information  applicable  to  OT&E  objectives 
and  to  assess  the  readiness  of  the  system  for 
operational  testing. 

c.  Insure  that  the  OT&E  is  effectively  planned  and 
conducted. 

d.  Bring  directly  to  the  attention  of  its  Military 
Service  Chief  and  Service 

Secretary,  or  Defense  Agency  Director  those  i 

program  issues  which  impact  adversely  on  the 
accomplishment  of  adequate  OT&E. 

^ 


r 1 
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e.  Provide  to  its  Service  Systems  Acquisition  Review 
f,  Council  (Sections  IV. B. 2.  of  reference  (e)  and  III.l. 

of  this  directive)  an  assessment  as  to  the  results 
and  adequacy  of  operational  testing  accomplished  to 
date  and  the  adequacy  of  operational  testing  planned 

for  the  future  to  support  the  Council's  recommendations.  3 

3.  In  addition,  each  DoD  Component  will  provide  within  its  I 

immediate  headquarters  staff  a strong,  full-time,  focal  ' 

point  organization  to  assist  the  independent  OT&E  field 

agency  to  keep  its  Military  Service  Chief  and  Service 
Secretary,  or  Defense  Agency  Director  fully  informed  as 
to  OT&E  needs  and  accomplishments. 

4.  Test  planning  for  DT&E  and  OT&E  should  be  coordinated 

at  the  test  design  stages  so  that  each  test  cycle  requires 
minimum  resources  and  yields  the  maximum  data  to  satisfy 
the  common  needs  of  the  material  developing  agency  and 
the  OT&E  agency.  Such  coordination  should  also  preclude 
unnecessary  duplication.  Development  testing  and  opera- 
tional testing  may  be  combined  where  clearly  identified 
and  significant  cost/time  benefits  would  result,  or  where 
separation  would  cause  delay  Involving  unacceptable  military 
risk  or  an  unacceptable  increase  in  the  acquisition  cost 
of  the  system.  When  combined  testing  is  conducted,  the 
necessary  test  conditions  and  test  data  required  by  both 
the  developing  agency  and  the  OT&E  agency  must  be  realized. 

Therefore,  the  developing  agency,  through  coordination  with  ' 


the  OT&E  agency,  must  Insure  that  the  combined  Lest  is  so 
planned  and  executed  as  to  p’-ovlde  the  necessary  operational 
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test  information  and  that  the  OT&E  agency  is  afforded 
the  opportunity  to  participate  actively  in  the  test. 

The  OT&E  agency  shall  provide  a separate  evaluation  of 
the  resultant  operational  test  information.  As  a normal 
practice  the  operational  tests  supporting  a production 
decision  will  be  conducted  independently  by  the  OT&E 
agency. 

5.  Acquisition  programs  shall  be  structured  so  that  opera- 
tional testing  is  commenced  as  early  as  possible  in  the 
development  cycle.  As  a minimum,  an  initial  phase  of 
operational  test  and  evaluation  (lOT&E)  will  be  accom- 
plished prior  to  the  first  major  production  decision 
adequate  to  provide  a valid  estimate  of  expected  system 
operational  effectiveness  and  operational  suitability. 
Preproduction  prototypes  will  be  employed  for  lOT&E  if 
they  are  reasonably  representative  of  the  expected  pro- 
duction items  and  allow  a valid  estimate  to  be  made  of 
expected  system  operational  effectiveness  and  operational 
suitability;  otherwise,  pilot  production  items  will  be 
employed  for  lOT&E. 

6.  For  complex  systems,  additional  phases  of  OT&E  may  be 
required  and  performed  with  pilot  or  preproduction  items 
subsequent  to  the  first  major  production  decision  but  prior 
to  the  availability  of  first  production  items.  Wlien  pro- 
duction items  are  available  in  sufficient  quantity,  follow- 
on  phases  of  OT&E  adequate  to  meet  the  full  objective 
outlined  above  will  be  accomplished  by  the  appropriate  DoD 
Component's  independent  OT&E  agency. 


1 
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7.  For  those  systems  which  have  an  interface  with  equip- 

^ ment  of  another  DoD  Component,  or  may  be  acquired  by 

two  or  more  Components,  joint  OT&E  will  be  conducted 
where  required.  Such  joint  testing  will  include  parti- 
cipation and  support  by  all  affected  Components  as 
appropriate.  An  independent  evaluation  will  be  submitted 
by  the  OT&E  agency  of  each  participating  Component. 

8.  When  required,  and  as  initiated  by  the  Deputy  DDR&E(T&E) , 
Joint  Operational  Test  and  Evaluation  (JOT&E)  programs 
will  be  conducted  to  evaluate  the  effectiveness  of 
weapon  systems,  or  of  comparative  weapon  system 
technologies,  under  as  realistic  operational  conditions 
as  possible.  Responsibility  for  managing  the  practical 
aspects  of  each  JOT&E  will  be  delegated  to  a specific 
Component,  with  supporting  forces  and  materiel  as 
required  from  other  Components.  When  feasible,  JOT&E 
objectives  will  be  accomplished  in  conjunction  with 
Service  or  JCS  scheduled  exercises. 

D.  Test  and  Evaluation  for  Major  Ships  of  a Class.  The  long 
design,  engineering,  and  construction  period  of  a major 
ship  will  normally  preclude  completion  of  the  lead  ship 
and  accomplishment  of  test  thereon  prior  to  decision  to 
proceed  with  follow  ships.  In  lieu  thereof,  successive 
phases  of  DT&E  and  OT&E  will  be  accomplished  as  early  as 
practicable  at  test  Installations  and  on  the  lead  ship 


so  as  to  rapidly  reduce  risks  and  thereby  minimize  the 
need  for  modification  to  follow  ships. 
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1.  When  combat  system  complexity  warrants,  there  will 
be  constructed  a combat  system  test  installation 
wherein  the  weapon,  sensor,  and  information  pro- 
cessing subsystems  are  integrated  through  their 
interfaces  in  the  manner  expected  in  the  ship  class. 
Adequate  initial  DT&E  and  OT&E  of  the  integration  of 
those  subsystems  will  be  accomplished  thereon  prior 

to  the  first  major  producton  decision  on  follow  ships. 
To  the  degree  practicable  first  generation  subsystems 
will  have  been  approved  for  service  use  prior  to  the 
initiation  of  integrated  operational  testing.  VHiere 
subsystems  cannot  be  service  approved  prior  to  the 
initial  operational  testing,  their  integration  will 
be  tested  at  the  test  site  installation  as  early  as 
possible  in  their  acquisition  cycle. 

2.  For  new  ship  types  incorporating  major  technical 
advancements  not  earlier  proven  in  hull  or  non-nuclear 
propulsion  design,  a prototype  incorporating  these 
advancements  will  be  employed.  If  the  major  techno- 
logical advancements  are  contemplated  in  only  some 
features  of  the  hull  or  non-nuclear  propulsion  design, 
the  test  installation  need  incorporate  only  the 
applicable  new  features.  Adequate  test  and  evaluation 
on  such  prototype  will  be  completed  prior  to  the  first 
major  production  decision  on  follow  ships. 
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3.  The  prototyping  of  Navy  nuclear  propulsion  plants 
will  be  accomplished  in  accordance  with  the  methods 
in  use  by  the  Energy  Research  and  Development 
Administration  (ERDA) . Construction  of  the  lead 
and  follow  ships  will  be  done  in  the  sequence  now 
being  used. 

4.  For  all  new  ship  classes,  continuing  phases  of  OT&E 
on  the  lead  ship  will  be  conducted  at  sea  as  early 
in  the  acquisition  process  as  possible  for  specified 
systems  or  equipments  and,  if  required,  full  ship 
operational  evaluation  to  the  degree  feasible. 

5.  A description  of  the  subsystems  to  be  included  in 
any  test  site  or  test  prototype,  the  schedules  to 
accomplish  test  and  evaluation,  and  any  exceptions 

to  the  above  policies  will  be  set  forth  in  the  initial 
and  any  subsequent  DCPs  and  approved  by  the  Secretary 
of  Defense. 

E.  Test  and  Evaluation  for  Onc-of-a-Kind  Systems.  For 

one-of-a-kind  systems,  or  systems  involving  procurement 
of  only  a very  few  over  an  extended  period,  the  principles 
of  DT&E  of  components,  subsystems,  and  prototype  or  first 
production  models  of  the  system  will  be  applied.  Com- 
patibility and  Interoperability  with  existing  or  planned 
equipments  will  be  tested.  OT&E  will  be  conducted  as 
early  as  possible  by  the  OT&E  agency  as  necessary  to 
provide  a valid  estimate  of  operational  effectiveness 


and  operational  suitability. 
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F.  Production  Acceptance  Test  and  Evaluation  (PAT&E) . PAT&E 
is  test  and  evaluation  of  production  items  to  demonstrate 
that  the  items  procured  fulfill  the  requirements  and 
specifications  of  the  procuring  contract  or  agreements. 

It  is  the  responsibility  of  each  DoD  Component  to 
accomplish  the  necessary  PAT&E  throughout  the  production 
phase  of  the  acquisition  process. 

G.  T&E  Master  Plan  (TEMP) . The  DoD  Component  shall  prepare 
as  early  as  possible  in  the  acquisition  process  but  no 
later  than  120  days  following  the  Demonstration  and 
Validation  Decision  (Milestone  I) , an  initial  version  of 
an  overall  test  and  evaluation  plan.  This  broad  plan 
should  identify  and  integrate  objectives,  responsibilities, 
resources,  and  schedules  for  all  T&E  to  be  accomplished 
prior  to  the  subsequent  key  decision  points.  The  TEMP 
will  be  kept  current  by  the  DoD  Component.  All  test  and 
evaluation  activities  as  prescribed  by  the  TEMP  shall 
comply  with  environmental  issues  in  accordance  with 
reference  (c) . The  TEMP  shall  include  consideration  of 
electromagnetic  compatibility  in  accordance  with 
reference  (d) , 

H.  Changes  to  Test  and  Evaluation  Plans.  The  DoD  Component 
will  insure  that  any  changes  made  in  test  plans  after 
approval  will  be  properly  documented  to  show  the  reasons 
for  change  and  the  approving  authority. 


" ■ I iwp^ipp^iMP I .1  - 
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I . Defense  Systems  Acquisition  Review  Council  (DSARC)/  | 

Service  Systems  Acquisition  Review  Council  ((S)SARC)/  | 

Decision  Coordinating  Paper  (DCP)  Procedures  for  Major  j 

Defense  Systems.  The  DSARC/ (S) SARC/DCP  procedures  for  | 

the  acquisition  of  major  defense  systems  are  described  ! 

in  detail  in  reference  (e) . The  following  paragraphs 
of  this  section  amplify  and  supplement  those  portions 
of  the  reference  which  pertain  to  Test  and  Evaluation. 

1,  The  DCP  prepared  for  use  at  the  time  of  the  Demon- 
stration  and  Validation  Decision  (Milestone  I)  for 
a major  Defense  System  will  identify  the  critical 
issues  and  areas  of  risk  to  be  addressed  by  test 
and  evaluation.  It  will  also  provide  the  test 
objectives  and  measures  of  effectiveness  related 
to  satisfaction  of  mission  need,  and  a sununary  of 
schedules  and  resource  requirements  applicable  to 
test  activity  prior  to  Milestone  II.  The  DSARC, 
and/or  (S)SARC,  in  its  review  will  determine  the 
adequacy  of  the  critical  issues,  test  objectives^ 
and  test  schedules. 

2.  When  the  DoD  Component  proposes  to  initiate  Full- 
Scale  Engineering  Development,  the  revised  DCP  will 
give  the  results  of  T&E  accomplished  to  date;  an  updated 
statement  of  critical  issues,  test  objectives,  and  areas 

of  risk  needing  further  assessment;  a summary  of  performance 


criteria  goals  and  thresholds;  and  an  overview  of  test 
plans,  milestones,  and  program  interrelationships. 
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The  cognizant  Component  shall  make  available 
supporting  details  of  test  plans  and  test  results 
as  requested  by  the  Deputy  DDR&E(T&E) (Section  VI). 

The  DSARC  and/or  (S)SARC  will  assess  and  comment 
to  the  Secretary  of  Defense  as  to  the  adequacy  of 
T&E  progress  and  of  T&E  planned  to  occur  prior  to 
the  first  major  production  decision. 

3.  The  DSARC  and/or  (S)Si\RC  in  its  review  prior  to 
the  first  major  production  decision  will  assess 
the  results  of  testing  to  date  and  plans  for  future 
testing.  To  support  a decision  to  proceed  with  production, 
it  will  comment  to  the  Secretary  of  Defense  as  to  the 
adequacy  of  these  test  results. 

4.  In  case  of  DCF  revisions  and  DSARC  or  (S)SARC  reviews 
subsequent  to  the  first  major  production  decision, 

an  updated  assessment  of  test  results,  plans,  obje.ctives, 
and  schedules  for  additional  test  and  evaluation  will 
be  provided, 

5.  The  DSARC  and  (S)SARC  are  supported  at  milestone  reviews 
respectively  by  the  Deputy  DDR&E(T&E)  and  by  Service 
development  and  OT&E  agencies,  who  provide  assessments 
as  to  the  results  and  adequacy  of  testing  accomplished 
to  date  and  the  adequacy  of  testing  planned  for  the 
future  to  support  the  Councils'  recommendations.  In 
addition,  the  Deputy  DDR&E(T5.E)  provides  directly  to 
the  Secretary  of  Defense  his  independent  assessment. 

This  assessment  is  attached  to  the  report  of  the  DSARC 
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Chairman  in  the  case  of  DSARC  reviews,  or  to  the 
assessment  of  the  Defense  Acquisition  Executive 
when  a DSARC  review  is  not  held. 

IV.  WAIVERS 

Waiver  of  the  accomplishment  of  the  T&E  as  outlined  in  the 

/ 

approved  DCF  will  be  granted  only  by  the  Secretary  of  Defense. 

V.  EXCLUSION’S 

Test  and  evaluation  of  nuclear  weapons  subsystems  which 
are  governed  by  other  joint  DoD/ERDA  agreements  are 
excluded  from  the  foregoing  provisions  of  this  directive. 

VI . RESPONSIBILIIIES  OF  THE  DEPUTY  DIRECTOR  OF  DEFENSE  RESEARCH 
AX’D  ENGINEERING,  TEST  AND  EVALUATION 

The  Deputy  DDR&E(Tf<E)  has  across-the-board  responsibility 
for  OSD  in  tost  and  evaluation  matters.  This  responsibility 
incl udcs : 

A.  Reviewing  test  and  evaluation  policy  and  procedures 
applicable  to  the  Department  of  Defense  as  a whole  and 
recommending  changes  he  believes  appropriate  directly 
to  the  Secretary  of  Defense 

B.  Monitoring  closely  the  test  and  evaluation  planned  and 
conducted  by  t'ne  DoD  Components  for  major  acquisition 
programs  and  for  such  other  programs  as  he  believes 
necessary . 

C.  Assisting  in  the  preparation  of,  and/or  reviewing,  the 
Test  and  Evaluation  Sections  of  DCPs  and  Program 


Memoranda  (PMs) . 
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D.  For  major  programs  under  DSARC  review,  reporting  to  the 
DSARC  and  the  Worldwide  Military  Command  and  Control  System 
Council,  as  appropriate,  and  directly  to  the  Secretary  oC 

Defense  his  assessment  as  to  the  results  and  adequacy  of 

i 

the  testing  accomplished  to  date  and  the  adequacy  of 
testing  planned  for  the  future  to  support  the  recommenda- 
tions of  the  DSARC.  This  independent  assessment  will 
be  forwarded  to  the  Secretary  of  Defense  attached  to  the 
report  of  the  DSARC  Chairman. 

E.  For  major  programs  under  (S)SARC  review,  participating 
in  the  OSD  review  of  (S)SARC  reports  to  the  Secretary 
of  Defense  and  reporting  to  the  Secretary  of  Defense 
his  independent  assessment,  attached  to  the  assessment 
of  the  Defense  Acquisition  Executive. 

F.  Monitoring  closely  such  joint  testing  as  is  accomplished 
by  the  DoD  Components  in  connection  with  their  planned 
acquisition  of  specific  systems.  In  addition,  initiating 
and  coordinating  the  accomplishment  of  such  additional 
joint  testing  as  is  necessary,  with  specific  delegation 
to  an  appropriate  Component  (or  Components)  of  all 
practical  aspects  of  the  joint  test. 


Initiating  and  coordinating  Joint  Operational  Test  and 
Evaluation  (JOT&E)  programs.  In  addition,  monitoring 
the  conduct  of  the  JOT&Es  and  reviewing  the  JOT&E  final 


reports . 
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H.  Coordinating  and  reviewing  the  test  and  evaluation  of 
foreign  systems  for  possible  DoD  or  NATO-wide  use. 

I.  Fulfilling  OSD  responsibilities  for  the  Major  Range 
and  Test  Facility  Base  (MRTFB)  in  accordance  with 
reference  (f) . 

J.  Monitoring,  only  to  the  extent  required  to  determine 
the  applicability  of  results  to  weapon  system  acquisi- 
tion or  modification,  that  test  and  evaluation: 

1,  Directed  by  the  Joint  Chiefs  of  Staff  which 
relates  to  the  Single  Integrated  Operational  Plan 
(STOP)  operational  factors. 

2.  Conducted  primarily  for  development  or  investigation 
of  organizational  or  doctrinal  concepts. 

To  accomplish  these  duties,  statements  of  critical  issues 
and  test  obiectlves,  test  plans,  and  test  results  will  be 
made  available  to  the  Deputy  DDR&ECT&E)  as  earlv  as  developed 
for  such  major  or  other  programs  as  he  may  reauest. 

VI I . REPORT! NG  REQUl RElPiNTS 

The  reporting  requirements  prescribed  herein  are  exempt 
from  formal  approval  and  control  in  accordance  with 
DoD  Directive  5000.19,  Enclosure  3,  Section  VI I. C. 

VIII  . EFFECTIVE  DATE  AND  IMi’LE.*'n:N’TAT10N 

This  Directive  is  effective  immediately.  Each  DoD  Component 
which  has  authority  and  responsibilities  under  reference  (b) 
will  review  its  documents  which  implement  this  Directive  and 
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will  forward  to  the  Director  of  Defense  Research  and 
Engineering  within  120  days  three  copies  of  each  document 
published  or  revised  as  a result  of  this  Directive. 
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